Dynamical fluctuations for semi-Markov processes 
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We develop an Onsager-Machlup-type theory for nonequilibrium semi-Markov pro- 
cesses. Our main result is an exact large time asymptotics for the joint probability of 
the occupation times and the currents in the system, establishing some generic large 
deviation structures. We discuss in detail how the nonequilibrium driving and the 
non-exponential waiting time distribution influence the occupation-current statistics. 
The violation of the Markov condition is reflected in the emergence of a new type of 
nonlocality in the fluctuations. Explicit solutions are obtained for some examples of 
driven random walks on the ring. 
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I. INTRODUCTION 

Stochastic processes enter physics because of some reduced or incomplete description 
in terms of variables whose states at earlier times do not uniquely determine their future 
states. Unless we go to infinite scale separations and treat the relevant set of variables as 
for example in the hydrodynamic limit, we expect that the reduced description allows for 
fluctuations and randomness in the dynamics. We then speak about the mesoscopic level 
of description for which reproducibility which is so typical for the macroscopic world, is not 
available yet. 

Furthermore, the resulting or effective descriptions in terms of stochastic dynamics 
cannot always be reliably treated within the Markov approximation. That Markovian level 
would also require particular time-scale separations such that the random evolution becomes 
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essentially memoryless. One easily looses the Markov property when the landscape of states 
and their mutual connections are getting very complicated and some further coarse-graining 
collects various states into one, if only to simplify things. For example in the theory of spin 
glasses, by combining various local minima of the free energy in one state and depending on 
the various activation energies, one could effectively obtain stretched exponential waiting 
time distributions. Or, for transport in strongly disordered systems, one can think that 
conduction is the result of a large series of hops, and the effective or total hopping rate must 
convolute different exponentials, cf. Mott's variable range hopping [25]. For biophysical 
processes such as in molecular motors or in ion channels one observes gate states that are 
either at the beginning or at the end of a series of states that appear different only in some 
minor (internal) rearrangement of molecules. The passage from entrance to exit through 
these internal and largely hidden states can again give rise to nonexponential waiting time 
distributions. See [17, 27, 29] for specific biomolecular realizations. In the examples that 
follow in section V, we present a very simple scenario of such a transport problem. The 
natural probabilistic environment is then that of semi-Markov processes, [16, 24]. Other 
scenario's can be due to the random nature of energy levels such as conjectured in blinking 
quantum dots for which the phenomenology suggests power law waiting time distributions, 
[6]. 



The purpose of the present paper is to investigate the role of entropy fluxes and of 
dynamical activity in the fluctuation theory for semi-Markov processes. Such a theory has 
been investigated for Markov jump and Markov diffusion processes in the light of recent 
studies in nonequilibrium statistical mechanics, [5, 9, 10, 19]. It remains important to 
characterize the fluctuation functionals in their physical role away from the strict Markovian 
context. Going now beyond these Markov processes, we are especially interested in the 
influence and in the role of the waiting time distribution in the dynamical fluctuations. 
After all, it parameterizes a time-symmetric factor in the transition events, whose influence 
on the joint occupation and current statistics needs to be understood. In this sense, we 
go here also beyond previous approaches (e.g. [1]), where only the time-antisymmetric 
factor is considered. Our results give detailed expressions for the fluctuation functionals 
and we interpret the role of the waiting time distribution in them. We provide some more 
results in the regime of small fluctuations, where we prove that the current and occupation 
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fluctuations decouple close to equilibrium. Here we also conclude that occupation fluc- 
tuations are in a sense more sensitive to non-Markovian behaviour than current fluctuations. 

The next section contains a brief introduction to the world (and the notation) of semi- 
Markov processes. Further elements of the semi-Markov theory are recalled in the Appendix. 
We then move to a general introduction on dynamical fluctuation theory, which contains 
our main formulations of the fluctuation functionals for semi-Markov processes. Subsec- 
tions are devoted to some corollaries and to the interpretation of these functionals. That 
includes a fluctuation symmetry for the entropy production and in section IV the treatment 
of small fluctuations around a nonequilibrium state. We end in Section V with examples of 
semi-Markovian transport in rings for which our results are getting fully explicit (for small 
fluctuations). In a more specific example we also compute the generating function and the 
first few cumulants for the current fluctuations. 

II. SEMI-MARKOV PROCESS 

A. Definitions and notation 

We consider jump processes on a finite space Q with states denoted by x,y, . . . The 
updating is time-homogeneous and in continuous time. Semi-Markov processes are non- 
Poissonian with a renewal property. This means that the probability of a jump from x to y 
at a certain time depends only on the states x, y and the time t since the last jump occurred. 
More precisely, let us denote Q(x,y,t) for the density of random transitions at time t from 
the state x to y. This so called semi-Markov kernel defines the process. Further it is useful 
to introduce 

r+oo 

Qfat) = Y]Q(x,y,t), p{x,y)= Q(x,y;t)dt (1) 
y Jo 

which are respectively the waiting (or sojourn) time distribution in x and the transition 
probabilities regardless of the waiting time, and 

/+oo 
Q(x;r)dr (2) 

the probability that the system rests at state x for at least time t. In the sequel we always 
assume that Q(x,y;t) = 0(t~ 3 ~ e ), e > asymptotically for t — > +oo so that the first and 
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the second moments with respect to the distributions Q(x, y; t) are finite. An important role 
plays the effective escape rate X(x) from x defined as the reciprocal to the average sojourn 
time in x, i.e., 

r-\-oo r-\-oo 

— = / rQ(x;r)dr= / A(x; r) dr (3) 
K x > Jo Jo 

We say that the semi-Markov process enjoys time-direction independence when 

Q(x, y; t) = p(x, y) Q(x; t) (4) 

That is the case when the waiting time only depends on the present (and not on the future) 
state. We make that assumption (4) throughout the paper. See section (VA) and (50) for 
a specific (counter)example. 
If the process is Markov then 

Q(x, y; t) = p(x, y) \{x) e" A ^ , A(x; *) = e" A ^ 

with X(x) the escape rate from state x; the product w(x,y) = X(x)p(x,y) is called the 
transition rate. The effective escape rates are X(x) = X(x). 

Further elements of the theory of semi-Markov processes are summarized in the Appendix. 
In particular, there we review a derivation of the (generalized) Master equation, and its 
formulation from the point of view of an embedded Markov chain. More details are of 
course available in the literature, see e.g. [24, 27] for physics introductions. 



B. Semi- Markov statistics 

We add here some ingredients of the theory of semi-Markov processes that relate to the 
statistical mechanics we are going for in the next section. 

As explained in the Appendix, see (A6) and Section A3, the stationary distribution p 
solves the stationarity equation 

^2jp(x, y) = , 3 P (x, y) = p(x)X(x)p(x, y) - p(y)\(y)p(y, x) ( 5 ) 
y 

where j p {x,y) is the stationary (probability) current from state x to y. Remark that this 
equation coincides with the stationarity condition for a continuous time Markov process 
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with transition rates X(x)p(x,y). In particular, the stationary distribution and currents 
depend on the waiting time distributions only through the effective escape rates. 

In this paper we go beyond the above stationary characterization of the semi-Markov 
process and we want to understand the structure of fluctuations of the occupations and 
currents around their stationary values, i.e., to develop a dynamical fluctuation theory for 
these processes. For that we need more details about the process and a natural starting 
point is the path-space distribution evaluating the plausibility of trajectories. This was the 
general idea of Onsager and Machlup, [26]. Here we follow the strategy developed in [22] 
for Markov systems. 

A path uj = (x t )o<t<T specifies the sequence of states together with the jump times, 

[Xq, ti, X\, t 2 , . . . , x n _i, tn,Xn), 0<t!<t 2 <...t n <T, 71=1,2,... (6) 

In order to construct a transient semi-Markov process started from a given initial distribution 
fi at time zero, we make a "stationarity" assumption about the history of the process in 
negative times: we let the age of an initial configuration sampled from \x be random and 
conditionally distributed according to the stationary process. Then the resulting path space 
distribution Pyiyj) giving the probability of a path uj has the density 

dP^u) = fi(x ) \(x ) A(x , xi) ti) Q{xx,x 2 ; t 2 -t 1 )... 

. . . Q(x n -i,x n ; t n - t„_i) A(x n ; T - t n ) dti . . . dt n (7) 

in which A(xo,xi;ti) = J^°° Q(xq, x±; t) dr is the waiting time distribution for the initial 
interval [0,ti], taking into account the random (stationarily distributed) age of the initial 
configuration Xq at time zero when the process starts. Similarly, the last term, A(x n ; T — t n ), 
comes out by integrating the waiting time distribution Q(x n ;t n+ i) over all possible times 
of the first jump outside the time interval, t n+ i > T. Note that for \i = p the above 
construction yields a stationary process. 

The apparent similarity between the first and the last terms in (7), representing the past 
and the future of the process, will be exploited next in the analysis of the time-reversal 
symmetry and its breaking. 
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C. Time-reversal and local detailed balance 



Any open system weakly coupled to its environment and being in thermal equilibrium 
with the latter has to satisfy two general conditions that directly follow from first principles: 
(i) its stationary distribution has the canonical form 

p{x) = \e-^ (8) 

with U the energy of the system and f3 the bath temperature, and (ii) the (effective) 
stochastic dynamics of the system is symmetric under time-reversal. This symmetry can be 
broken either by starting the dynamics from a nonstationary condition (i.e., in transients), 
or provided the system is coupled to several thermal reservoirs mutually not in thermal 
equilibrium (i.e., in transport processes). Then the dynamics of the system is no longer 
time-reversal symmetric, however, this symmetry is broken in a very specific way: a natural 
'measure' of irreversibility, see below, coincides with the change of entropy together in the 
system and in the environment. For a general argument see, e.g., [20]. Next we specify 
these considerations to the semi-Markov processes. 



As a standard measure of irreversibility of the process we consider the path-dependent 
quantity defined as the relative plausibility of a path with respect to its time-reversed coun- 
terpart. Introducing the time-reversal 9uj of a path uj as {9uj) t = our-t or 

0u) = (x n , T - t n , x n -i, T - . . . , xi, T - tx, x ) (9) 

cf. (6), we define 



fi(x )\(x )A(x , xi,ti)... Q(x n _i,x n ; t n - t n _i)A(x„; T - t n ) 

= log = 

fi T (x n )\(x n )A(x n , x n -i, T -tn) ... Q(xi, x ; t 2 - ti)A(x Q ; h) 

where [it is the distribution at time T as evolved from the [i at time zero, i.e., the solution 
of the generalized Master equation (A6) or (A8). As we restrict ourselves to the case of 
time-direction independence (4), equation (10) considerably simplifies to 

S» = log/^o) -l0gM*n) + El0g A ^-. l) ^- 1 ' X ; ) (11) 



i=l 
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which is formally the same as one has for a Markov process with escape rates X(x). 

Assume first that the system is in thermal equilibrium with a heat bath. Then p = p is 
given by (8) and, by time-reversibility, S p (u) = 0, pathwise. This is equivalent to 

X(y)p(y,x) 

which is a generalized detailed balance condition (recall that in the Markov case, w(x, y) = 
X(x)p(x,y) are transition rates). Using (5), this is further equivalent to the absence of 
all stationary currents, j p {x,y) = 0. A rigorous argument for the equivalence between the 
(generalized) detailed balance condition and the time-reversibility of time-direction inde- 
pendent semi-Markov processes can be found in [8, 29]. Remark that the assumption of 
time-direction independence is crucial here and cannot be easily abandoned. 

To make a step beyond thermal equilibrium, observe first that the right-hand side of (12) 
reads (3 times the heat flux (= entropy flux) into the heat bath, per a single transition 
x — > y in the system. This is clearly a global condition since all the local entropy fluxes 
derive from a potential (or state quantity) f3U. However, it has a natural local variant that 
only requires that 

log =7-7^7—^-7 = entropy flux (x -> y) (13) 
X{y)p(y,x) 

no matter whether the entropy fluxes derive from a potential or not. Physically this corre- 
sponds to a system coupled to several heat baths on nonequal temperatures, assuming that 
each transition x — > y is assisted by no more than one reservoir. The condition (13) is 
called local detailed balance. When modeling a particular physical process, we usually take 
the individual entropy fluxes per each transition as a priori known, cf. [3] . 

Under the local detailed balance condition, the path quantity S p is the sum of two terms: 
of the difference — log/iT^n) + log/^(xo) which is to be understood as the variable entropy 
increase in the system (one checks that its expectation equals the increase in Shannon 
entropy), and of the total entropy flux into environment. The latter adds up contributions 
from all transitions along the random trajectory. 



Let us conclude here with two remarks: first of all, local detailed balance is not a math- 
ematical condition but rather a general guiding principle to be followed when modeling an 
arbitrary open system driven out of equilibrium. It is mainly because of this that we restrict 
ourselves to time-direction independent processes. For time-direction dependent processes 
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the assumption of local detailed balance does not seem to make sense. Furthermore, a direct 
consequence of local detailed balance is a symmetry in the fluctuations of the time- integrated 
entropy flux and also of the time-integrated currents (fluctuation theorems). It follows easily 
from (11) via standard manipulations. See also [1, 14, 20] for another and more detailed 
approach. 



D. Resolution with respect to time-reversal 

By construction, the entropy flux is intimately related to the time-antisymmetric part 
in the logarithmic probabilities (or action). In order to separate more explicitly the 
time-symmetric sector of fluctuations from the time-antisymmetric one, we will make a 
parametrization, inspired by the Markov case [22]: there one relates the transition rates 
k(x,y) to an equilibrium reference process with rates ko{x,y) so that 

k(x,y) = k (x,y)e^ G ^ 

and G(x,y) = —G(y,x). (One checks that for a Markov processes such a representation 
always exists). By local detailed balance, G(x,y) can be seen as {(3 times) the work done 
by an extra (with respect to the reference) force along the transition x — > y. 

As a generalization, we consider as a suitable equilibrium reference system another semi- 
Markov process with the waiting time distributions Qo(x,y;t) = Qo(x;t)po(x,y) such that 
\og[p (x,y)/p (y,x)} derives from a potential (the global detailed balance condition) and 
that the waiting times of the original and the reference processes are related by [30] 

Q(x,y;r) = Q (x,y;r) e^ G ^ +A o^ (14) 

for some G(x,y) = —G(y,x). The term Ag(x) is a compensator fixed by the normalization 
condition 

POD 

/ drQ(x;r) = l (15) 
Jo 

Note that the parametrization (14) preserves the time-direction independence property. By 
comparing with condition (13) of local detailed balance, the entropy flux per transition 
x — > y reads 

X(x)p(x,y) = + _ 

\(y)p(y,x) 
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with some potential u that is explicitly computable. The process breaks the (global) detailed 
balance unless G(x, y) also derives from a potential. We can therefore say that G(x, y) is the 
forcing of the process. The decomposition of the entropy flux into potential and nonpotential 
parts is hence fixed by comparing to a particularly chosen reference equilibrium. 

Here is how we split the action into, respectively, a time-antisymmetric and a time- 
symmetric parts. From (7), the logarithmic density of our process with respect to the 
equilibrium reference process P° is 

log = \ E G(x t ., x t+ ) + £ dt A%(x t ) (17) 

where = denotes that we have only taken the time-extensive part, and neglected temporal 
boundary terms that will become redundant. In the first term the sum is over all jump 
times in u and by (16) it is equal to the total (i.e., time-integrated) entropy flux along path 
u, for the original process. The second term in (17) is manifestly time-symmetric and it 
can be understood as a time-undirected dynamical activity, or what we have called traffic 
in [19, 22] (relatively with respect to the reference dynamics.) 

The path-space average of (17) with respect to our process gives its dynamical or also 
called, Kolmogorov- Sinai entropy — the relative entropy between the process and its reference 
as distributions on paths. The decomposition (17) suggests to define two functionals which 
take averages over the two terms in (17) separately. They will appear later; for the stationary 
regime with stationary density p we divide (17) by T and let T — > +oo to write 

1 / 1 dp p\ S + 



with 



S = ^2p(x)\(x)p(x,y)G{x,y) = ^^2j p (x,y)G(x,y) 

x,y x,y 

T = 5>(s) A&(s) (18) 



The quantity S is the stationary average of the entropy flux per unit time. Note that it does 
not depend on the particular choice of the reference equilibrium process as it is insensitive to 
adding any potential difference to the driving G. The second component of the dynamical 
entropy, T, measures the stationary dynamical activity in the sense of its "excess" with 
respect to the equilibrium dynamics taken as a reference. 



10 



III. DYNAMICAL FLUCTUATIONS 
A. Scope 

Equilibrium statistical mechanics provides us with a fluctuation theory through which 
the thermodynamic potentials can also be understood as fluctuation functionals. Natural 
variables for these functionals usually are the energy and particle densities, the magnetization 
or still other characteristics of an equilibrium state. The precise formulation of all that is 
found in the theory of large deviations as pioneered by Boltzmann, Planck, and Einstein, 
which starts from the identification of the thermodynamic entropy with the logarithm of a 
probability, see e.g. [13, 18, 23]. We call this a static fluctuation theory where the main 
extensive parameter is the spatial volume or the number of particles. That also has an 
extension to spatially extended systems out of equilibrium but there is no simple way of 
determining the stationary distribution. The difficulties with its direct determination can be 
overcome by analyzing typical paths along which macroscopic fluctuations get spontaneously 
created, via exploiting methods of analytical mechanics. The fluctuation functionals, often 
called nonequilibrium free energies, are then found to solve an appropriate Hamilton- Jacobi 
equation [4]. 

In contrast, dynamical fluctuation theory deals with deviations from stationary behavior 
that are observed over a large time period. This formulation is most useful for mesoscopic 
systems in contact with large external reservoirs, where these fluctuations can be visible 
on the level of the system. An immediate consequence is the variational characterizations 
of the steady state regime, much as the Gibbs variational principle characterizes thermal 
equilibrium from the minimum of a free energy functional. Beyond that, the question is 
once more whether the corresponding (now, dynamical) fluctuation functionals allow for 
a natural physical interpretation, whether they can provide relations between quantities 
directly accessible via measurement etc. Indeed, we recall that in equilibrium the Onsager- 
Machlup theory constructs actions for the distribution of macroscopic histories that relate 
to response coefficients and to dissipation functions, [26]. It would be most interesting to 
obtain an extension of these functionals to reach domains further away from equilibrium 
and also in situations different from those of fluctuating hydrodynamics. 

Here we consider the set-up of semi- Mar kovian jump processes; the Markov case has been 
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discussed in [22]. The questions can however be put in a more general context, as now follows. 

We consider a path (or history or trajectory) and we observe the occupation of states and 
the various transitions over states in some large time-interval [0, T]. More precisely, we first 
look at the fraction of time that the system spends in a state x G f2 for one specific path uj\ 

tPr(x) := ^ [ 5[x t = x] dt (19) 
1 Jo 

where 5[xt — x] is 1 whenever xt = x and zero otherwise. Obviously, p>t{x),x G f2, defines 
a probability law but it is itself random as dependent on the stochastic trajectory u. We 
assume that these paths are drawn from the unique steady state with stationary density p. 
Then, as time T — > +oo we have convergence of p-rix) to that p(x); that corresponds to 
an assumption of ergodicity. Secondly we define the empirical densities of jumps between 
states x and y: 

k T (x, y):=-^2 <%t- = x] 5[x t+ = y] (20) 

t<T 

where the sum is over all jump times and the configurations before and after the 

jump, respectively. Again, that is a random quantity, typically converging for large T to 
p(x)X(x)p(x,y). Finally there is the empirical current 

jr(x,y) = k T (x,y) - k T (y,x) (21) 

The question of our dynamical fluctuation theory is to see and to physically understand the 
asymptotic statistics for //t(^) and jx{x,y). What values do these assume and with what 
probability? In other words, we take a probability law p, on Q and a family j = (j(x,y)) 
with ^2 y j(x, y) = and we ask for the steady state probability 

P[^ T ~ p ; j T ~j] oce- T/( ^ (22) 

as T — ► +oo, to realize these /i and j along the trajectories. We already suggest here that 
there exists a rate function /(/i, j) which exactly picks up the leading order in T. That 
rate function is the Legendre transform of the log-generating function of the occupation and 
current statistics which would give more direct access to the various cumulants, but we will 
not need these here. Our ambition here is not so much on the computational but rather 
on the conceptual level, to understand what is the generic structure of I(fj,,j) as well as 
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its possible physical configuration. In particular, we want to stress the similarities and the 
differences with Markov processes, and to recognize the influence of modifying the waiting 
time distribution. As we will see explicitly in the examples of section V, both the current and 
the occupation statistics do pick up also higher moments of the waiting time distribution, 
yielding markers for non-Markovian behavior. 



The fluctuation functionals appearing as rate functions such as the in the exponent 

of (22), are understood as relative entropy densities, see [11, 13, 28] for an introduction to 
the systematic theory of large deviations. The relative entropy is between a modified and 
the original process where the modified process is chosen such as to make the deviations 
typical. In our case, we deal with temporal processes and the relative entropy density is like 
the rate of change of dynamical entropies between the two processes. More specifically, to 
compute P[//, j] := P[yUy — A* Jt — j], we define a new semi-Markov process in the following 
way: 



where F(x,y) = —F(y,x), with J °° Q*(x; r)dr = 1 and J2 y P*( x ^y) = 1- Comparing with 
(14) we see that Q* is of the form 



Hence, A° F+G (x) = A G (x) + A(x). Most important now, we require that \x and j are 
stationary in this new (modified) dynamics, i.e., 



B. Joint occupation-current statistics 




p*(x,y) = Z(x)p(x,y)ei 



Q*(x;r)p*(x,y) = Q(x;r)p(x,y)e l 2 F ^+ A ^ 



(23) 



j(x, y) = fj,(x)X*(x)p*(x, y) - /i(y)X*(y)p*(y, x) 



(24) 



In terms of these new quantities the joint fluctuation functional reads 




(25) 



13 



Indeed, by using the explicit form (17) of the path-space measure we get 



P[/*, j] = I dP»%T = ^Jt = J]= I dP;(Lu)^(Lu)5[^ T = p,jT = j] (26) 



where 



log -7— (w) = 2^10" 



^ - ti)p*(xi,x i+ i) 

n— 1 j 



-2 
i=i 

x,y x 

Note that we have only written the extensive part in time, because we are considering the 
large time limit anyway. Continuing the computation, we now see that: 

Pf/x, j] = e -$?^, v j(*,vM*.v)-TE*ti*M*) j dP;(u)5[fi T = fijr = j] (27) 

Finally, in the large time limit we have that J dP*' T {oj)5[^t = V*,3t — j] ~ 1, because \i 
and j are typical in the modified process. Therefore the probability of the fluctuations has 
the asymptotic form: 

F[/iJ] oce- TI ^ (28) 

with given in (25). 



C. Occupation statistics 

A subquestion concerns the time-symmetric fluctuation sector; to understand the statis- 
tics of the occupations alone. That means to look at (19) and to write similarly to (22), 

¥[p T ~ n] oc e ~ TIM (29) 

To find the fluctuation functional /(//) of only the occupation statistics, one can perfectly 
repeat the argument for occupation-current statistics above, but this time it suffices to 
restrict oneself to the class of modified processes driven by gradient forces, F(x, y) = V(y) — 
V(x). Again, the point is that the potential V can be chosen such that it makes /i typical. 
The result for the occupation fluctuation functional is 

10*)= M^)A(x) (30) 
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We see that A appears as the quantity to average over with //, for expressing the rate at 
which the system deviates from the \i— statistics. We can interpret A(x) as the excess traffic 
of the modified process that makes /i, j typical, with respect to the original process: 

J(/i) = fa) A(x) = - t G {n) (31) 

X 

where the traffic functional is defined as 

t G (/i):=^/i(^)A° G (x) (32) 

X 

to be compared with (18). Using (15), we can deduce the response relation 

-^3n,G{x,y) (33) 



dG{x,y) 2 
with 

3v,g(x, y) = yu(x)A(x)p(x, y) - (i(y)X(y)p(y, x) (34) 

the expected transient current for a fluctuation \i in a dynamics as in (14) determined by G. 
In this sense, the traffic can be understood as a potential with respect to the currents; cf. 
similar remarks for the Markov processes, [19, 22]. However, in contrast with the latter, the 
traffic (32) does not allow for a simple "kinematic" interpretation in terms of an expected 
number of jumps irrespectively of their direction. 

D. Fluctuation symmetry 

We now turn to the antisymmetric fluctuation sector, where we recover the fluctuation 
theorem, cf. [1, 14]. This is a direct consequence of the local detailed balance we have 
proposed in section II: Indeed, local detailed balance dictates that (the extensive part of) 
the entropy production is 

s {u) = ^G(x t _,x t+ ) = - ^2j T (x,y)G(x,y) (35) 

t<T x,y 

where jx is again the empirical current. As a consequence: 
F[j T ~ j] = / dPH S[j T = j] 
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which means that 

niT^j] fs.,,^,^,,,) (36) 

again asymptotically for T — > +00, i.e., up to temporal boundary terms. Indeed and we 
already concluded in (16), because of local detailed balance, G(x,y) is the generalized ther- 
modynamic force for the transition x — ► y in the entropy production. Taking the logarithm 
and the limit T — > +00, formula (36) establishes a symmetry in the dynamical fluctuations 
of the entropy production, cf. [20]. 

For convenience, we add in the next section some more details concerning the physi- 
cal interpretation of various players in the dynamical fluctuation theory, along with the 
comparison to the Markov case. 



E. Comparison with Markov processes 

In the fluctuation functionals two new quantities appear: A(x) and F(x, y). The meaning 
of F(x,y) can be made clear through local detailed balance (13). We already know that for 
the original process, local detailed balance means that 

X(y)p(y,x) 

where a(x, y) = G(x, y) + u(x) — u(y) is the entropy flux between system and reservoir per 
jump from x to y, see (16). For the modified process we find 

\*(x)p*(x,y) = eaiX) y )+aex{Xjy) , 3g x 

\*(y) p *(y iX ) 

where now a ex (x,y) is the excess entropy flux of the modified process with respect to the 
original process and is given by 

/ \ ^ \ , fZ(x)X*(x)\ , fZ{y)\*{y)\ , , 

,.(*,„) =*■(*,„)+ log (^LU j _ log ( j (39) 

This means that, again up to some 'potential difference,' (the last two terms in (39)) the 
term F(x, y) is the extra force one adds to the system to make fi and j stationary. For a 
Markov process this extra 'potential' becomes zero. 
When the process is not Markov we still have that 

X] y) F fa y) = ^j(x, y)a ex (x, y) (40) 



x,y x,y 



16 



whenever the stationarity condition 'Yl iy j{x^y) = is fulfilled. In particular, this means 
that the potential terms do not make a time-extensive contribution to the total entropy flux 
since along any trajectory, the empirical currents jT{x,y) always satisfy that stationarity 
condition up to corrections 0(1/T). In this sense, the second term in the joint fluctuation 
functional functional can really be called an excess entropy flux. 



We have mentioned already above how the quantity A(x) appears. In the case of a 
Markov process, A(x) is simply (minus) the excess escape rate of the modified process with 
respect to the original process: 

A(x) = A(x) - A*(x) (41) 

Combining (14) with (15), the Ag(a;) can be expanded around a Markov reference: e.g., by 
writing 

Q (x,y;t) = w(x,y)e- x ^ t J2< 



n=0 



Then, the normalization (15) leads to 



+oo 

5>(s,y)e* G <**> = (A(x) - A°(x)) QT 

y ra=0 



oW(i) 



(A(x) - A%(x)r 



n=0 x 



a: 



Assuming that a^°'(x) = 1 and that the other a^ n '(x), n > 1 are all small gives rise to an 
expansion of A^(x) and of A(x) around the Markov case (41). 



IV. SMALL FLUCTUATIONS 



In this section we examine the regime of small (or Gaussian) fluctuations, to say more 
about the traffic and the influence of the waiting time distributions on the fluctuations. For 
this we make a quadratic approximation to the various functionals but not (necessarily) 
around equilibrium. 
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A. General 

We let the fluctuations in the empirical distribution and the empirical current be param- 
eterized with some e, 

fi(x) = p(x)[l + e/ii(x)] 

where p(x) is the stationary measure, and j p (x, y) is the stationary current. Also A(x) and 
F(x,y) are now of order e and to indicate that, we replace those with eA(x) and eF(x,y). 
To first order in e, the conditions (23)-(24) become 



A(x) 



2(r) 

where 



(x <— > y)' (42) 
^ (43) 



1 Z" 00 
(^=1^)' <^>, = / o drr 2 g(a;;r) (44) 

The fluctuation functional within the quadratic approximation is, cf. (25), 



f 2 



I(fi,j) =e 2 ^p(x)/ii(x)A(x) + -^h{x,y)F{x,y) (45) 

x x,y 

We observe that for small fluctuations, only the first and second moments of the waiting 
time distributions contribute. Furthermore, the second term on the right hand side of (42)) 
marks the difference with the Markov case since for the exponentially distributed waiting 
times (t 2 ) x = 2 (t) 2 ,. Due to the presence of this term beyond Markov, the functional I(fi,j) 
no more splits into a sum over different transitions, hence, it is responsible for the emergence 
of a new type of nonlocality in the fluctuations, not present under the Markov condition. 

B. Close to equilibrium 



In the case of a (global) detailed balance dynamics, we can prove that the occupation 
and current fluctuations become decoupled within the quadratic approximation. Indeed, by 
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using the detailed balance condition, the equation (42) can be explicitly solved for the extra 
forcing F: 



F(x, y) = ^(y) - fr(x) + ( (r2) - 2 (r) ' ) A(s) - (^Vy^) A(y) 



x \ I y 



(A 



(r) 
p(x)p{x,y) 

Substituted in the equation (43), it yields 



(46) 



(47) 

where we have used that Yl y h( x iV) = 0- Clearly, that A only depends on the occupations 
fii and not on the currents. That is why the occupation and the current fluctuations become 
statistically independent, with the joint fluctuation functional being a sum of the occupation 
functional and the current functional: J(/i, j) = I(fi) + where 

J(/i)=e 2 ^p(x)/i 1 (x)A(x) (48) 

X 

and 



J(j) = jE 1 w J i(x,y) 2 (49) 



x,y 

Suppose now that the (global) detailed balance is slightly broken, in the following sense: 
take p(x,y) = po(x,y) + epi(x,y), where po(x,y) are detailed balanced transition prob- 
abilities. As the functionals (48)-(49) are already of order e 2 , the small deviation from 
detailed balance will not contribute. We have therefore still uncorrelated statistics for the 
occupations and currents in the close-to-equilibrium regime, and the marginal fluctuation 
functionals (48)-(49) remain unchanged. 

Remark that whereas the current statistics has exactly the same form as for a Markov 
process with escape rates X(x) = 1/ (t) , we observe a difference in the occupation statistics. 
As the difference between semi-Markov and Markov processes lies in the time-symmetric part 
of the path-space probabilities, it should come as no surprise that the occupation statistics 
are more sensitive to details of the waiting time distribution than are the current statistics. 
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V. EXAMPLES 

First we give an example of semi-Markov process obtained from a Markov model via 
coarse-graining. Its generalized version is then used to illustrate our dynamical fluctuation 
theory. 

A. Semi- Markov from Markov 

Consider a Markov random walk as in Fig. 1, with three kinds of states x^y^ and for 
% — 1, . . . , N on a ring (N +1 = 1). 




FIG. 1: Markov random walk with 'hidden' states. 



As the arrows in Fig. 1 suggest, the only transitions allowed 
Hi — > a?i+i, and i, whereas all the others are are forbidden. This is a model of a 

one-dimensional random walk with 'hidden' states: the particle at state Xi can go 'to the 
right' (from X{ to through the 'hidden' state yi only, and 'to the left' through Z{ only. 

More specifically, we fix A^, X y , X z > and we set the Markov transition rates to 

W(xi -> yi) = p\ x 
W(xi -> Zi) = q\ x 

W(Vi -> X i + l) = Xy 

W(zi -* Xi-i) = X z 

for some p + q = 1. The rates do not depend on the position % on the ring. 

Since every pair of 'hidden' states y^ and z% have the unique precursor Xi, we can follow 
a simple coarse-graining procedure: for each % to take together all three states Xj, y^ and 
Zi, see Fig. 2. By construction, the coarse-grained random walk on the new 'block' states, 
denoted by q, is a semi-Markov process. 

We can calculate the probability that the walker occupies q for a time r > t before 
jumping to c i+ i, and its time derivative is the density Q(cj, c i+ i, t). Exploiting that the only 
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X 



FIG. 2: Coarse-graining of the states. 

possibility of going from q to c i+ i (or Cj_i) is via the 'hidden' state iji (or Zj), the waiting 
time distributions (1) read 



Q( Ci ,c i+1 ;t) = [ drp\ x e- x * T ■ Xye-^-^ 
Jo 



V K A, 



-Xxt 



Q(ci,Ci_i]t) = qX x X 



X X — Xy 

g — A 2 t g— \ x t 



X x — X z 

It is clear that Q(cj, Cj + i; t) and Q(cj, q_i; t) determine the dynamics of the new stochastic 
process which is semi-Markov: the updates are decided by the immediate history but the 
waiting time distribution is not exponential and it can depend on the specific transition. 
Observe also that 

p+oo r+oo 

/ dtQ(d,c i+1 ;t) =p, / dtQ(c i ,c i - 1 ]t) = q 
Jo Jo 

which indicates a driving whenever p ^ q. In all events the stationary distribution is 
uniform over the ring. 



Let us investigate two different limits of this example. First let X x — * +oo. In this limit 

we get 

Q(d, c m ; t) = pX y e~ Xyt , Q(q, t) = qX z e~ Xzt (50) 

Unless X y = X z , the process is not Markov. In general it is semi-Markov with direction 
dependent waiting time distribution. 

Secondly we consider the limits X y — > X x and X z — * X x together. The resulting transition 
densities are 

Q{ Ci ,c i+1 ;t) =pX 2 x te~ Kt , Q^c^t) = q X 2 X t e~ Kt (51) 
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Here the time dependence of the two clocks for going to the left or to the right are the same. 
In particular, it does not matter whether the walker first picks a direction or just picks the 
first clock that rings. Still it is not a Markov process, because the waiting time distributions 
of the clocks are not exponential. This process is semi-Markov with waiting time-direction 
independence. 



B. CTRW on the ring 

The semi-Markov model obtained in the previous section is an example of continuous 
time random walk (CTRW). In this section we give some explicit solutions to equations that 
appear in dynamical fluctuation theory, for a CTRW on the ring. 

We continue with states that represent the sites on a ring of length N, with translation 
invariance as in the above explicit example. Let the transition densities be 

Q(x,t) = Q(t), p{x, x + 1) = p, p(x + l,x) — q 

We restrict to the small fluctuations as in Section IV. Because the current fluctuation 
j = j p + eji has to satisfy J2 y j( x i v) = 0; we see that ji(x, x + 1) + ji(x, x — 1) = 0, and 
therefore ji(x, x + 1) = j± is a constant on the ring. 
The equations (42) become 

ji = -r^Y-r [pfii{x) - qni(x + 1) - AF{x + 1, x + 2) + BF(x, x + 1) - AF{x - 1, x)} 
N (r) 

A(x) = — rT (qF(x-l,x)-pF(x,x + l)) (52) 
2(r) 

where the constants A and B are 



A = pq(y 



(r 2 )_ 
2 



2<T>' 

(p 2 + q 2 ) (r 2 ) (p - qf 



B - 

2(r) 2 



The fluctuation functional reads 

IfaJ) = ^E^(^) A ^) + e ^Y, F ^ x + !) ( 53 ) 

X X 

The second term on the right-hand side is easily computed by summing (52) over all x. 
Using that J2 x Hi(x) = we get 
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Defining now f(x, x + 1) by F(x, x + 1) — f(x,x + l) + N JiVl , we see that ^ x f(x, x + 1) = 0, 
hence, / is of the gradient form f(x, x + 1) — V(x + 1) — V(x). Substituting in (52), we get 

= ■^rr\ptii{x)-qtii{x + l)-Af{x + l,x + 2)+Bf{x,x + l)-Af{x-l,x)] 
N (r) 

1 N 2 (r) jx 

A ( x ) = 7rrr(lf( x _ -Pf( x ' x + !)) + (9 -P)" 



2(r)^ J v ' y " v ' ^ ^ 5-2A 

iV 2 It) 7i 

=: A'(x) + (g-p)^f|^ (54) 
Finally, (53) takes the form 

= /(//) + /(j) 

= ^E^)A'W + g^| (55) 

So there is a decoupling of current and occupation statistics. Whereas, in general, this 
occurs only for small fluctuations and close to equilibrium, here it is apparently valid 
arbitrarily far from equilibrium. The fundamental reason lies in the translation-invariance 
property of the dynamics. 



Let us further examine the statistics of current fluctuations, exploiting that the functional 
is explicit. Using that in the Markov case 2{B — 2 A) = 1, we can write in general 
= C -1 /m(j)) with Im(J) ^ ne fluctuation functional for a Markov process with the same 
average waiting time (r), and with C given by 

C = l + (2 P -l)*(^M-l) (56) 

where Var(r) = (r 2 ) — (r) 2 is the variance of the waiting time distribution. Thus C is a 
correction factor with respect to the Markov case; one checks that C > 0. We also see that 
the bigger the variance of the waiting times, the smaller C _1 , and therefore the will 
become flatter. Remark that in detailed balance (i.e. for p = ~), we have that C — 1. So 
in this case one has the same fluctuation functional as in the Markov it should 

be according to (48). Furthermore, there is a fundamental difference between the cases 
Var(r) > (r) 2 and Var(r) < (r) 2 (i.e., whether the variance of the waiting times is bigger or 
smaller than in the Markov case). When the variance is smaller than in the Markov case, 
C as a function of p tends to get bigger when p gets closer to or 1. In the other case, C 
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becomes smaller when p is closer to or 1. 

Finally we can compute the occupation statistics I{p). Formulae considerably complicate 
however when N is large and that is why we choose to be explicit only for N = 3 (a ring 
with three sites). Similarly as for we can write I(p) = C'~ l I m{h) , with 

Im((j) = J2(pni(x) - qnx(x - l)) 2 

3(r) ^ 

2e 2 



X 

-2 



(i + p 2 - p)[^(iy + ^(2) 2 - mi)/^)] 



3(r) 

using that //i(l) + A*i(2) + /^i(3) = 0. The C is given by 

C-l + (/- P+ l)«-l) (57) 

Also here, the bigger the variance of the waiting times, the flatter the fluctuation functional 
becomes. However, in the detailed balance case the functional does not reduce to the Markov- 
form. This is indeed what we concluded in the discussion after (48). There is another 
difference with the current fluctuations: as a function of p, the fluctuation functional gets 
flatter when p gets closer to or 1, independent of the sign of Var(r)/ (r) 2 — 1. 

C. Generating function approach to current statistics 

Finally we consider (arbitrary) current fluctuations on the ring and compare two possible 
approaches. As explained in the present paper, one way of computing this is via contraction 
of the joint fluctuation functional: 

I(j)=MI(fM,j) (58) 

As is however often the case, explicit computations proceed more easily via the generating 
function 

G(v) = (e TvjT ) (59) 

Defining g(v) = lim^^oo ^ log G(v), one can prove that g(v) exists and is the Legendre 
transform of the fluctuation functional 

g(v) = sup{vj - (60) 

j 
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and vice versa. It generates the current cumulants, see e.g. [2, 7, 9] for applications to 
nonequilibrium interacting particle systems. By using the Laplace transform as in (A7), 
with Q(s) the Laplace transform of Q(r), we solve the equation 

Q(s*) = (61) 

K ' pe v + qe~ v 

for s*; then g{v) = s*. 

As an example, consider the waiting time distribution 

Q(r) = T^f^^ ( 62 ) 



for A > 0, a > 1. (Note that a = 1 represents the Markov case.) For this distribution 

m = ( T ) = T ° 2 = Var(r) = 

A A z 

A x a 



and (61) becomes 



^ s *)=(aT^) = (Pe" + 90" 1 (63) 



or, 



= A^e" + ge"")" - A (64) 

By taking derivatives at v = we obtain the next explicit expressions for the current 
moments: 

p-q 

Jp = 

m 

Uik-3 P f) = -(- 2 -l)(p -Q? + - 
1 m m z m 

U{j T -j P f) = -(- 2 -i)(^- 2 -2)( P - q r + \ 3 -4-2)( P - q ) 

1 m m z m A m m A 

It is interesting to see that the variance of the currents picks up a term depending on the 
driving p — q and that this term is only non-zero if the process is non-Markov. This is indeed 
what we expect from the discussions about the small fluctuations regime, where we found 
that in, or close to, the detailed balance case the current fluctuations are quite insensitive 
to changes in the variance of the waiting times. 
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APPENDIX A: MORE BACKGROUND ON SEMI-MARKOV PROCESSES 

We give some more details on the structure of semi-Markov processes. See [24, 27] for 
excellent introductions. 

1. Embedded Markov chain 

To have a good intuition about a semi-Markov process it is worth seeing it as a discrete 
time Markov chain to which specific waiting times are added. In particular, the discrete 
sequence of states in the semi-Markov process is drawn from the Markov chain with transition 
probabilities p(x,y): if at time Tq the state Xq was created, then we choose the next state 
X\ with probability p(xq,Xi). Secondly, we have to decide how long has been the waiting 
between the creation of Xq and the creation of X\. For that we need the waiting time 
distribution: let its distribution (i.e. the integrated probability density) be 1 — A ^ y ^ , 
so that the (total) distribution function for the random waiting time r% — tq between the 
creation of xq and the creation of x\, is p(xo, x±) — A(x , x\\t). 

A semi-Markov process is thus constructed most elegantly from a Markov renewal process 
(x n , r n ) n >o for x n G Q and To < T\ < . . . < r n < . . . denoting jump times, with transition 
probabilities 

P[Zra+l = V, r n+l -T n <t\x n , T n , X n _i, r n _i, . . . , Xq, Tq] = p(x n , fj) ~ A(x n , y;t), U,t>0 

satisfying all of (1) with Q(x,y;t) = —4:A(x,y;t). 

The process (x n ) with transition probabilities p(x, y) is called the embedded Markov chain. 
For the randomness in time, we define the jump counting process v(t),t > of the total 
dynamical activity up to time t, 

v(t) = sup{n > : r n < t} 

which we assume is finite with probability one. The semi-Markov process corresponding to 
that renewal process is then defined by 

x(t) = x„(t), t>0 
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where the state at time t is just equal to x n of the embedded Markov chain, if v(t) = n. For 
a Markov process u(t) is just a Poisson process. 

2. Master equation 

An intuitive way of deriving a generalization of the master equation for semi-Markov 
processes is by considering the corresponding Markov process (xt, T t )o<t<T, where xt is the 
configuration of the semi-Markov process at time t and r t is the time that the system has 
been in this configuration since its last jump. 

With this in mind we can write down the transition probabilities for the process (x t , r t ), 

P(x, t; y, t) = Prob(x t+d t = y, r t+dt = r'\x t =x,r t = r) (Al) 

We then have that with X(x;r) = Q(x;t)/A(x;t),A(x;t) = ^ y A(x, y; r), X(x, y; r) = 
Q(x,y,T)/A(x,y;r), 

i) P(x, r; x, r + dt) = 1 - X(x, r)dt + o(dt); 

ii) P{x,T]y,0) = X{x,y,T)dt + o(dt). 

and other transition probabilities are of order o(dt) . 

Let us now look at the evolution of probability densities /i t (x,T) in this dynamics. It is 
easily seen that for r ^ 0: 

Ht+dt (x, t) = fit(x, t - dt) [1 - dt X(x; r - dt)] (A2) 



from which it follows that 

dfi t {x,T) _ dfi t (x, r) 
dt ~ dr 

and for r = 0, 



- IM(x, t)X(x;t) (A3) 



/■oo 

°)= / drJ2^(y,r)X(y,x;r) (A4) 
Jo y 

Because we are mainly interested in the process xt (which is no longer Markov), we integrate 
over the waiting times r: fo(x) = J °° dr/j, t (x, r). Doing that for (A3) we get (assuming that 
/i t (x, oo) = 0): 

= Ht(x, 0) - l°° fi t (x, t)X(x; r) dr (A5) 
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Using (A4) we thus arrive at a generalized Master equation: 

= / drJ^[^ t (j/,T)A(2/,2;;T) - ^ t (x } r)X(x,y;r)] = -^j t {x,y) (A6) 
v y 

which also defines the currents j^(x,y). 

There is obviously a downside to this equation, and that is that we need knowledge of 

Ht(x, t) to compute the time derivative of [At{x). There is a way to circumvent this problem 

by using the Laplace transform. 



Taking the Laplace transform of a function / as 

CO 



/(*)= / f(t)e~ st dt (A7) 



and putting ^q(x, t) = fio(x)X(x)A(x, r) it can be proven that 

oo pT / pT 

dr/i T (x,r)A(x,y;r) = / ^(x) ip(x,y;T-t)dt+/j, (x) (X(x)p(x,y) - dtip(x,y;t) 
o Jo V Jo 

with (p having Laplace transform 



tp[x,y; s) 



A(x; s) 

We substitute that in the master equation: 

= ^/ dt i\Vt(y) ~ tM)(y)]<p(y,x;T -t) - [n t {x) - fj, (x)]<p(x,y;T -t)} 

+ [My)Ky)p(y, x ) - tM>(x)\(x)p(x, y)] (as) 

y 

The function <p represents the memory of the semi-Markov process. The faster <p(x, y\ t) 
decays to zero, the less memory we have. For example, for a Markov process with escape 
rates A(x) we find that tp(x, y; t) = X(x)p(x, y) S(t). Indeed a Markov process has no memory 
(and the master equation for Markov processes arises as a special case of this generalized 
equation). 



3. Stationarity 

Stationarity of the semi-Markov process means that path-space averages of time- 
independent observables are still time independent. But as above we can use the corre- 
sponding Markov process (x t , r t )o<t<T- Stationarity of the semi-Markov process is ensured 
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by demanding that /i f (x, r) := p(x, r) is stationary. Solving then (A3) with the LHS zero, 
and using p(x) = J °° drp(x,r), we get p(x,r) = p(x)X(x)A(x;r). This means that if we 
know that the system is in a configuration x, then the probability that it has been there 
already for a time r is equal to X(x)A(x; r). 

The stationary measure of our semi-Markov process is p(x) oc jHr where 7r(x) is the 
stationary measure of the embedded Markov chain. We write 

l7r(a;) 

p[x) 



£A(x) 

for some normalization £, the overall average waiting time. In this notation, we see that the 
stationary currents are 

jp (x, y) = <*)P^V)-<V)PM (A9) 

and that they are zero iff the embedded Markov chain is detailed balance. That however 
is only equivalent with time-reversal invariance if the semi-Markov process is time-direction 
independent, see e.g. [8, 27, 29]. 
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